Resource provisioning systems and methods

ABSTRACT

A method and apparatus managing a set of processors for a set of queries is described. In an exemplary embodiment, a device receives a set of queries for a data warehouse, the set of queries including one or more queries to be processed by the data warehouse. The device further provisions a set of processors from a first plurality of processors, where the set of processors to process the set of queries, and a set of storage resources to store data for the set of queries. In addition, the device monitors a utilization of the set of processors as the set of processors processes the set of queries. The device additionally updates a number of the processors in the set of processors provisioned based on the utilization/Furthermore, the device processes the set of queries using the updated set of processors.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/905,599, filed Jun. 18, 2020, entitled “Resource Provisioning Systemsand Methods,” which is continuation of U.S. patent application Ser. No.16/378,371, filed Apr. 8, 2019, entitled “Resource Provisioning Systemsand Methods,” which is a continuation of U.S. patent application Ser.No. 14/518,826, filed Oct. 20, 2014, entitled “Resource ProvisioningSystems and Methods,” now issued as U.S. Pat. No. 10,325,032 whichclaims the benefit of U.S. Provisional Application Ser. No. 61/941,986,entitled “Apparatus And Method For Enterprise Data Warehouse DataProcessing On Cloud Infrastructure,” filed Feb. 19, 2014, thedisclosures of which are incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The present disclosure relates to resource management systems andmethods that manage resources related to data processing and datastorage.

BACKGROUND

Many existing data storage and retrieval systems are available today.For example, in a shared-disk system, all data is stored on a sharedstorage device that is accessible from all of the processing nodes in adata cluster. In this type of system, all data changes are written tothe shared storage device to ensure that all processing nodes in thedata cluster access a consistent version of the data. As the number ofprocessing nodes increases in a shared-disk system, the shared storagedevice (and the communication links between the processing nodes and theshared storage device) becomes a bottleneck that slows data read anddata write operations. This bottleneck is further aggravated with theaddition of more processing nodes. Thus, existing shared-disk systemshave limited scalability due to this bottleneck problem.

Another existing data storage and retrieval system is referred to as a“shared-nothing architecture.” In this architecture, data is distributedacross multiple processing nodes such that each node stores a subset ofthe data in the entire database. When a new processing node is added orremoved, the shared-nothing architecture must rearrange data across themultiple processing nodes. This rearrangement of data can betime-consuming and disruptive to data read and write operations executedduring the data rearrangement. And, the affinity of data to a particularnode can create “hot spots” on the data cluster for popular data.Further, since each processing node performs also the storage function,this architecture requires at least one processing node to store data.Thus, the shared-nothing architecture fails to store data if allprocessing nodes are removed. Additionally, management of data in ashared-nothing architecture is complex due to the distribution of dataacross many different processing nodes.

The systems and methods described herein provide an improved approach todata storage and data retrieval that alleviates the above-identifiedlimitations of existing systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present disclosureare described with reference to the following figures, wherein likereference numerals refer to like parts throughout the various figuresunless otherwise specified.

FIG. 1 is a block diagram depicting an example embodiment of the systemsand methods described herein.

FIG. 2 is a block diagram depicting an embodiment of a resource manager.

FIG. 3 is a block diagram depicting an embodiment of an executionplatform.

FIG. 4 is a block diagram depicting an example operating environmentwith multiple users accessing multiple databases through multiplevirtual warehouses.

FIG. 5 is a block diagram depicting another example operatingenvironment with multiple users accessing multiple databases through aload balancer and multiple virtual warehouses contained in a virtualwarehouse group.

FIG. 6 is a block diagram depicting another example operatingenvironment having multiple distributed virtual warehouses and virtualwarehouse groups.

FIG. 7 is a flow diagram depicting an embodiment of a method formanaging data storage and retrieval operations.

FIG. 8 is a flow diagram depicting an embodiment of a method forprovisioning new virtual warehouses.

FIGS. 9A and 9B depict a flow diagram of an embodiment of a method forscaling resources.

FIG. 10 is a flow diagram depicting an embodiment of another method forscaling resources.

FIG. 11 is a block diagram depicting an example computing device.

DETAILED DESCRIPTION

The systems and methods described herein provide a new platform forstoring and retrieving data without the problems faced by existingsystems. For example, this new platform supports the addition of newnodes without the need for rearranging data files as required by theshared-nothing architecture. Additionally, nodes can be added to theplatform without creating bottlenecks that are common in the shared-disksystem. This new platform is always available for data read and datawrite operations, even when some of the nodes are offline formaintenance or have suffered a failure. The described platform separatesthe data storage resources from the computing resources so that data canbe stored without requiring the use of dedicated computing resources.This is an improvement over the shared-nothing architecture, which failsto store data if all computing resources are removed. Therefore, the newplatform continues to store data even though the computing resources areno longer available or are performing other tasks.

In the following description, reference is made to the accompanyingdrawings that form a part thereof, and in which is shown by way ofillustration specific exemplary embodiments in which the disclosure maybe practiced. These embodiments are described in sufficient detail toenable those skilled in the art to practice the concepts disclosedherein, and it is to be understood that modifications to the variousdisclosed embodiments may be made, and other embodiments may beutilized, without departing from the scope of the present disclosure.The following detailed description is, therefore, not to be taken in alimiting sense.

Reference throughout this specification to “one embodiment,” “anembodiment,” “one example” or “an example” means that a particularfeature, structure or characteristic described in connection with theembodiment or example is included in at least one embodiment of thepresent disclosure. Thus, appearances of the phrases “in oneembodiment,” “in an embodiment,” “one example” or “an example” invarious places throughout this specification are not necessarily allreferring to the same embodiment or example. In addition, it should beappreciated that the figures provided herewith are for explanationpurposes to persons ordinarily skilled in the art and that the drawingsare not necessarily drawn to scale.

Embodiments in accordance with the present disclosure may be embodied asan apparatus, method or computer program product. Accordingly, thepresent disclosure may take the form of an entirely hardware-comprisedembodiment, an entirely software-comprised embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,embodiments of the present disclosure may take the form of a computerprogram product embodied in any tangible medium of expression havingcomputer-usable program code embodied in the medium.

Any combination of one or more computer-usable or computer-readablemedia may be utilized. For example, a computer-readable medium mayinclude one or more of a portable computer diskette, a hard disk, arandom access memory (RAM) device, a read-only memory (ROM) device, anerasable programmable read-only memory (EPROM or Flash memory) device, aportable compact disc read-only memory (CDROM), an optical storagedevice, and a magnetic storage device. Computer program code forcarrying out operations of the present disclosure may be written in anycombination of one or more programming languages. Such code may becompiled from source code to computer-readable assembly language ormachine code suitable for the device or computer on which the code willbe executed.

Embodiments may also be implemented in cloud computing environments. Inthis description and the following claims, “cloud computing” may bedefined as a model for enabling ubiquitous, convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned via virtualization and released with minimalmanagement effort or service provider interaction and then scaledaccordingly. A cloud model can be composed of various characteristics(e.g., on-demand self-service, broad network access, resource pooling,rapid elasticity, and measured service), service models (e.g., Softwareas a Service (“SaaS”), Platform as a Service (“PaaS”), andInfrastructure as a Service (“IaaS”)), and deployment models (e.g.,private cloud, community cloud, public cloud, and hybrid cloud).

The flow diagrams and block diagrams in the attached figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flow diagrams or block diagrams may representa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It will also be noted that each block of the block diagramsand/or flow diagrams, and combinations of blocks in the block diagramsand/or flow diagrams, may be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flow diagram and/orblock diagram block or blocks.

The systems and methods described herein provide a flexible and scalabledata warehouse using a new data processing platform. In someembodiments, the described systems and methods leverage a cloudinfrastructure that supports cloud-based storage resources, computingresources, and the like. Example cloud-based storage resources offersignificant storage capacity available on-demand at a low cost. Further,these cloud-based storage resources may be fault-tolerant and highlyscalable, which can be costly to achieve in private data storagesystems. Example cloud-based computing resources are available on-demandand may be priced based on actual usage levels of the resources.Typically, the cloud infrastructure is dynamically deployed,reconfigured, and decommissioned in a rapid manner.

In the described systems and methods, a data storage system utilizes anSQL (Structured Query Language)-based relational database. However,these systems and methods are applicable to any type of database, andany type of data storage and retrieval platform, using any data storagearchitecture and using any language to store and retrieve data withinthe data storage and retrieval platform. The systems and methodsdescribed herein further provide a multi-tenant system that supportsisolation of computing resources and data between differentcustomers/clients and between different users within the samecustomer/client.

FIG. 1 is a block diagram depicting an example embodiment of a new dataprocessing platform 100. As shown in FIG. 1, a resource manager 102 iscoupled to multiple users 104, 106, and 108. In particularimplementations, resource manager 102 can support any number of usersdesiring access to data processing platform 100. Users 104-108 mayinclude, for example, end users providing data storage and retrievalrequests, system administrators managing the systems and methodsdescribed herein, and other components/devices that interact withresource manager 102. Resource manager 102 provides various services andfunctions that support the operation of all systems and componentswithin data processing platform 100. As used herein, resource manager102 may also be referred to as a “global services system” that performsvarious functions as discussed herein.

Resource manager 102 is also coupled to metadata 110, which isassociated with the entirety of data stored throughout data processingplatform 100. In some embodiments, metadata 110 includes a summary ofdata stored in remote data storage systems as well as data availablefrom a local cache. Additionally, metadata 110 may include informationregarding how data is organized in the remote data storage systems andthe local caches. Metadata 110 allows systems and services to determinewhether a piece of data needs to be accessed without loading oraccessing the actual data from a storage device.

Resource manager 102 is further coupled to an execution platform 112,which provides multiple computing resources that execute various datastorage and data retrieval tasks, as discussed in greater detail below.Execution platform 112 is coupled to multiple data storage devices 116,118, and 120 that are part of a storage platform 114. Although threedata storage devices 116, 118, and 120 are shown in FIG. 1, executionplatform 112 is capable of communicating with any number of data storagedevices. In some embodiments, data storage devices 116, 118, and 120 arecloud-based storage devices located in one or more geographic locations.For example, data storage devices 116, 118, and 120 may be part of apublic cloud infrastructure or a private cloud infrastructure. Datastorage devices 116, 118, and 120 may be hard disk drives (HDDs), solidstate drives (SSDs), storage clusters, Amazon S3™ storage systems or anyother data storage technology. Additionally, storage platform 114 mayinclude distributed file systems (such as Hadoop Distributed FileSystems (HDFS)), object storage systems, and the like.

In particular embodiments, the communication links between resourcemanager 102 and users 104-108, metadata 110, and execution platform 112are implemented via one or more data communication networks. Similarly,the communication links between execution platform 112 and data storagedevices 116-120 in storage platform 114 are implemented via one or moredata communication networks. These data communication networks mayutilize any communication protocol and any type of communication medium.In some embodiments, the data communication networks are a combinationof two or more data communication networks (or sub-networks) coupled toone another. In alternate embodiments, these communication links areimplemented using any type of communication medium and any communicationprotocol.

As shown in FIG. 1, data storage devices 116, 118, and 120 are decoupledfrom the computing resources associated with execution platform 112.This architecture supports dynamic changes to data processing platform100 based on the changing data storage/retrieval needs as well as thechanging needs of the users and systems accessing data processingplatform 100. The support of dynamic changes allows data processingplatform 100 to scale quickly in response to changing demands on thesystems and components within data processing platform 100. Thedecoupling of the computing resources from the data storage devicessupports the storage of large amounts of data without requiring acorresponding large amount of computing resources. Similarly, thisdecoupling of resources supports a significant increase in the computingresources utilized at a particular time without requiring acorresponding increase in the available data storage resources.

Resource manager 102, metadata 110, execution platform 112, and storageplatform 114 are shown in FIG. 1 as individual components. However, eachof resource manager 102, metadata 110, execution platform 112, andstorage platform 114 may be implemented as a distributed system (e.g.,distributed across multiple systems/platforms at multiple geographiclocations). Additionally, each of resource manager 102, metadata 110,execution platform 112, and storage platform 114 can be scaled up ordown (independently of one another) depending on changes to the requestsreceived from users 104-108 and the changing needs of data processingplatform 100. Thus, in the described embodiments, data processingplatform 100 is dynamic and supports regular changes to meet the currentdata processing needs.

During typical operation, data processing platform 100 processesmultiple queries (or requests) received from any of the users 104-108.These queries are managed by resource manager 102 to determine when andhow to execute the queries. For example, resource manager 102 maydetermine what data is needed to process the query and further determinewhich nodes within execution platform 112 are best suited to process thequery. Some nodes may have already cached the data needed to process thequery and, therefore, are good candidates for processing the query.Metadata 110 assists resource manager 102 in determining which nodes inexecution platform 112 already cache at least a portion of the dataneeded to process the query. One or more nodes in execution platform 112process the query using data cached by the nodes and, if necessary, dataretrieved from storage platform 114. It is desirable to retrieve as muchdata as possible from caches within execution platform 112 because theretrieval speed is typically much faster than retrieving data fromstorage platform 114.

As shown in FIG. 1, data processing platform 100 separates executionplatform 112 from storage platform 114. In this arrangement, theprocessing resources and cache resources in execution platform 112operate independently of the data storage resources 116-120 in storageplatform 114. Thus, the computing resources and cache resources are notrestricted to specific data storage resources 116-120. Instead, allcomputing resources and all cache resources may retrieve data from, andstore data to, any of the data storage resources in storage platform114. Additionally, data processing platform 100 supports the addition ofnew computing resources and cache resources to execution platform 112without requiring any changes to storage platform 114. Similarly, dataprocessing platform 100 supports the addition of data storage resourcesto storage platform 114 without requiring any changes to nodes inexecution platform 112.

FIG. 2 is a block diagram depicting an embodiment of resource manager102. As shown in FIG. 2, resource manager 102 includes an access manager202 and a key manager 204 coupled to a data storage device 206. Accessmanager 202 handles authentication and authorization tasks for thesystems described herein. Key manager 204 manages storage andauthentication of keys used during authentication and authorizationtasks. For example, access manager 202 and key manager 204 manage thekeys used to access data stored in remote storage devices (e.g., datastorage devices in storage platform 114). As used herein, the remotestorage devices may also be referred to as “persistent storage devices.”A request processing service 208 manages received data storage requestsand data retrieval requests (e.g., database queries). For example,request processing service 208 may determine the data necessary toprocess the received data storage request or data retrieval request. Thenecessary data may be stored in a cache within execution platform 112(as discussed in greater detail below) or in a data storage device instorage platform 114. A management console service 210 supports accessto various systems and processes by administrators and other systemmanagers. Additionally, management console service 210 may receiverequests from users 104-108 to issue queries and monitor the workload onthe system. In some embodiments, a particular user may issue a requestto monitor the workload that their specific query places on the system.

Resource manager 102 also includes an SQL compiler 212, an SQL optimizer214 and an SQL executor 210. SQL compiler 212 parses SQL queries andgenerates the execution code for the queries. SQL optimizer 214determines the best method to execute queries based on the data thatneeds to be processed. SQL optimizer 214 also handles various datapruning operations and other data optimization techniques to improve thespeed and efficiency of executing the SQL query. SQL executor 216executes the query code for queries received by resource manager 102.

A query scheduler and coordinator 218 sends received queries to theappropriate services or systems for compilation, optimization, anddispatch to execution platform 112. For example, queries may beprioritized and processed in that prioritized order. In someembodiments, query scheduler and coordinator 218 identifies or assignsparticular nodes in execution platform 112 to process particularqueries. A virtual warehouse manager 220 manages the operation ofmultiple virtual warehouses implemented in execution platform 112. Asdiscussed below, each virtual warehouse includes multiple executionnodes that each include a cache and a processor.

Additionally, resource manager 102 includes a configuration and metadatamanager 222, which manages the information related to the data stored inthe remote data storage devices and in the local caches (i.e., thecaches in execution platform 112). As discussed in greater detail below,configuration and metadata manager 222 uses the metadata to determinewhich data files need to be accessed to retrieve data for processing aparticular query. A monitor and workload analyzer 224 oversees theprocesses performed by resource manager 102 and manages the distributionof tasks (e.g., workload) across the virtual warehouses and executionnodes in execution platform 112. Monitor and workload analyzer 224 alsoredistributes tasks, as needed, based on changing workloads throughoutdata processing platform 100. Configuration and metadata manager 222 andmonitor and workload analyzer 224 are coupled to a data storage device226. Data storage devices 206 and 226 in FIG. 2 represent any datastorage device within data processing platform 100. For example, datastorage devices 206 and 226 may represent caches in execution platform112, storage devices in storage platform 114, or any other storagedevice.

Resource manager 102 also includes a transaction management and accesscontrol module 228, which manages the various tasks and other activitiesassociated with the processing of data storage requests and data accessrequests. For example, transaction management and access control module228 provides consistent and synchronized access to data by multipleusers or systems. Since multiple users/systems may access the same datasimultaneously, changes to the data must be synchronized to ensure thateach user/system is working with the current version of the data.Transaction management and access control module 228 provides control ofvarious data processing activities at a single, centralized location inresource manager 102. In some embodiments, transaction management andaccess control module 228 interacts with SQL executor 216 to support themanagement of various tasks being executed by SQL executor 216.

FIG. 3 is a block diagram depicting an embodiment of an executionplatform 112. As shown in FIG. 3, execution platform 112 includesmultiple virtual warehouses 302, 304, and 306. Each virtual warehouseincludes multiple execution nodes that each include a data cache and aprocessor. Virtual warehouses 302, 304, and 306 are capable of executingmultiple queries (and other tasks) in parallel by using the multipleexecution nodes. As discussed herein, execution platform 112 can add newvirtual warehouses and drop existing virtual warehouses in real timebased on the current processing needs of the systems and users. Thisflexibility allows execution platform 112 to quickly deploy largeamounts of computing resources when needed without being forced tocontinue paying for those computing resources when they are no longerneeded. All virtual warehouses can access data from any data storagedevice (e.g., any storage device in storage platform 114).

Although each virtual warehouse 302-306 shown in FIG. 3 includes threeexecution nodes, a particular virtual warehouse may include any numberof execution nodes. Further, the number of execution nodes in a virtualwarehouse is dynamic, such that new execution nodes are created whenadditional demand is present, and existing execution nodes are deletedwhen they are no longer necessary.

Each virtual warehouse 302-306 is capable of accessing any of the datastorage devices 116-120 shown in FIG. 1. Thus, virtual warehouses302-306 are not necessarily assigned to a specific data storage device116-120 and, instead, can access data from any of the data storagedevices 116-120. Similarly, each of the execution nodes shown in FIG. 3can access data from any of the data storage devices 116-120. In someembodiments, a particular virtual warehouse or a particular executionnode may be temporarily assigned to a specific data storage device, butthe virtual warehouse or execution node may later access data from anyother data storage device.

In the example of FIG. 3, virtual warehouse 302 includes three executionnodes 308, 310, and 312. Execution node 308 includes a cache 314 and aprocessor 316. Execution node 310 includes a cache 318 and a processor320. Execution node 312 includes a cache 322 and a processor 324. Eachexecution node 308-312 is associated with processing one or more datastorage and/or data retrieval tasks. For example, a particular virtualwarehouse may handle data storage and data retrieval tasks associatedwith a particular user or customer. In other implementations, aparticular virtual warehouse may handle data storage and data retrievaltasks associated with a particular data storage system or a particularcategory of data.

Similar to virtual warehouse 302 discussed above, virtual warehouse 304includes three execution nodes 326, 328, and 330. Execution node 326includes a cache 332 and a processor 334. Execution node 328 includes acache 336 and a processor 338. Execution node 330 includes a cache 340and a processor 342. Additionally, virtual warehouse 306 includes threeexecution nodes 344, 346, and 348. Execution node 344 includes a cache350 and a processor 352. Execution node 346 includes a cache 354 and aprocessor 356. Execution node 348 includes a cache 358 and a processor360.

In some embodiments, the execution nodes shown in FIG. 3 are statelesswith respect to the data the execution nodes are caching. For example,these execution nodes do not store or otherwise maintain stateinformation about the execution node or the data being cached by aparticular execution node. Thus, in the event of an execution nodefailure, the failed node can be transparently replaced by another node.Since there is no state information associated with the failed executionnode, the new (replacement) execution node can easily replace the failednode without concern for recreating a particular state.

Although the execution nodes shown in FIG. 3 each include one data cacheand one processor, alternate embodiments may include execution nodescontaining any number of processors and any number of caches.Additionally, the caches may vary in size among the different executionnodes. The caches shown in FIG. 3 store, in the local execution node,data that was retrieved from one or more data storage devices in storageplatform 114 (FIG. 1). Thus, the caches reduce or eliminate thebottleneck problems occurring in platforms that consistently retrievedata from remote storage systems. Instead of repeatedly accessing datafrom the remote storage devices, the systems and methods describedherein access data from the caches in the execution nodes which issignificantly faster and avoids the bottleneck problem discussed above.In some embodiments, the caches are implemented using high-speed memorydevices that provide fast access to the cached data. Each cache canstore data from any of the storage devices in storage platform 114.

Further, the cache resources and computing resources may vary betweendifferent execution nodes. For example, one execution node may containsignificant computing resources and minimal cache resources, making theexecution node useful for tasks that require significant computingresources. Another execution node may contain significant cacheresources and minimal computing resources, making this execution nodeuseful for tasks that require caching of large amounts of data. Yetanother execution node may contain cache resources providing fasterinput-output operations, useful for tasks that require fast scanning oflarge amounts of data. In some embodiments, the cache resources andcomputing resources associated with a particular execution node aredetermined when the execution node is created, based on the expectedtasks to be performed by the execution node.

Additionally, the cache resources and computing resources associatedwith a particular execution node may change over time based on changingtasks performed by the execution node. For example, a particularexecution node may be assigned more processing resources if the tasksperformed by the execution node become more processor intensive.Similarly, an execution node may be assigned more cache resources if thetasks performed by the execution node require a larger cache capacity.

Although virtual warehouses 302-306 are associated with the sameexecution platform 112, the virtual warehouses may be implemented usingmultiple computing systems at multiple geographic locations. Forexample, virtual warehouse 302 can be implemented by a computing systemat a first geographic location, while virtual warehouses 304 and 306 areimplemented by another computing system at a second geographic location.In some embodiments, these different computing systems are cloud-basedcomputing systems maintained by one or more different entities.

Additionally, each virtual warehouse is shown in FIG. 3 as havingmultiple execution nodes. The multiple execution nodes associated witheach virtual warehouse may be implemented using multiple computingsystems at multiple geographic locations. For example, a particularinstance of virtual warehouse 302 implements execution nodes 308 and 310on one computing platform at a particular geographic location, andimplements execution node 312 at a different computing platform atanother geographic location. Selecting particular computing systems toimplement an execution node may depend on various factors, such as thelevel of resources needed for a particular execution node (e.g.,processing resource requirements and cache requirements), the resourcesavailable at particular computing systems, communication capabilities ofnetworks within a geographic location or between geographic locations,and which computing systems are already implementing other executionnodes in the virtual warehouse.

Execution platform 112 is also fault tolerant. For example, if onevirtual warehouse fails, that virtual warehouse is quickly replaced witha different virtual warehouse at a different geographic location.

A particular execution platform 112 may include any number of virtualwarehouses 302-306. Additionally, the number of virtual warehouses in aparticular execution platform is dynamic, such that new virtualwarehouses are created when additional processing and/or cachingresources are needed. Similarly, existing virtual warehouses may bedeleted when the resources associated with the virtual warehouse are nolonger necessary.

In some embodiments, virtual warehouses 302, 304, and 306 may operate onthe same data in storage platform 114, but each virtual warehouse hasits own execution nodes with independent processing and cachingresources. This configuration allows requests on different virtualwarehouses to be processed independently and with no interferencebetween the requests. This independent processing, combined with theability to dynamically add and remove virtual warehouses, supports theaddition of new processing capacity for new users without impacting theperformance observed by the existing users.

FIG. 4 is a block diagram depicting an example operating environment 400with multiple users accessing multiple databases through multiplevirtual warehouses. In environment 400, multiple users 402, 404, and 406access multiple databases 414, 416, 418, 420, 422, and 424 throughmultiple virtual warehouses 408, 410, and 412. Although not shown inFIG. 4, users 402, 404, and 406 may access virtual warehouses 408, 410,and 412 through resource manager 102 (FIG. 1). In particularembodiments, databases 414-424 are contained in storage platform 114(FIG. 1) and are accessible by any virtual warehouse implemented inexecution platform 112. In some embodiments, users 402-406 access one ofthe virtual warehouses 408-412 using a data communication network, suchas the Internet. In some implementations, each user 402-406 specifies aparticular virtual warehouse 408-412 to work with at a specific time. Inthe example of FIG. 4, user 402 interacts with virtual warehouse 408,user 404 interacts with virtual warehouse 410, and user 406 interactswith virtual warehouse 412. Thus, user 402 submits data retrieval anddata storage requests through virtual warehouse 408. Similarly, users404 and 406 submit data retrieval and data storage requests throughvirtual warehouses 410 and 412, respectively.

Each virtual warehouse 408-412 is configured to communicate with asubset of all databases 414-424. For example, in environment 400,virtual warehouse 408 is configured to communicate with databases 414,416, and 422. Similarly, virtual warehouse 410 is configured tocommunicate with databases 416, 418, 420, and 424. And, virtualwarehouse 412 is configured to communicate with databases 416, 422, and424. In alternate embodiments, one or more of virtual warehouses 408-412communicate with all of the databases 414-424. The arrangement shown inFIG. 4 allows individual users to send all data retrieval and datastorage requests through a single virtual warehouse. That virtualwarehouse processes the data retrieval and data storage tasks usingcached data within one of the execution nodes in the virtual warehouse,or retrieves (and caches) the necessary data from an appropriatedatabase. The mappings between the virtual warehouses is a logicalmapping, not a hardware mapping. This logical mapping is based on accesscontrol parameters related to security and resource access managementsettings. The logical mappings are easily changed without requiringreconfiguration of the virtual warehouse or storage resources.

Although environment 400 shows virtual warehouses 408-412 configured tocommunicate with specific subsets of databases 414-424, thatconfiguration is dynamic. For example, virtual warehouse 408 may bereconfigured to communicate with a different subset of databases 414-424based on changing tasks to be performed by virtual warehouse 408. Forinstance, if virtual warehouse 408 receives requests to access data fromdatabase 418, virtual warehouse 408 may be reconfigured to alsocommunicate with database 418. If, at a later time, virtual warehouse408 no longer needs to access data from database 418, virtual warehouse408 may be reconfigured to delete the communication with database 418.

FIG. 5 is a block diagram depicting another example operatingenvironment 500 with multiple users accessing multiple databases througha load balancer and multiple virtual warehouses contained in a virtualwarehouse group. Environment 500 is similar to environment 400 (FIG. 4),but additionally includes a virtual warehouse resource manager 508 andmultiple virtual warehouses 510, 512, and 514 arranged in a virtualwarehouse group 516. Virtual warehouse resource manager 508 may becontained in resource manager 102. In particular, multiple users 502,504, and 506 access multiple databases 518, 520, 522, 524, 526, and 528through virtual warehouse resource manager 508 and virtual warehousegroup 516. In some embodiments, users 502-506 access virtual warehouseresource manager 508 using a data communication network, such as theInternet. Although not shown in FIG. 5, users 502, 504, and 506 mayaccess virtual warehouse resource manager 508 through resource manager102 (FIG. 1). In some embodiments, virtual warehouse resource manager508 is implemented within resource manager 102.

Users 502-506 may submit data retrieval and data storage requests tovirtual warehouse resource manager 508, which routes the data retrievaland data storage requests to an appropriate virtual warehouse 510-514 invirtual warehouse group 516. In some implementations, virtual warehouseresource manager 508 provides a dynamic assignment of users 502-506 tovirtual warehouses 510-514. When submitting a data retrieval or datastorage request, users 502-506 may specify virtual warehouse group 516to process the request without specifying the particular virtualwarehouse 510-514 that will process the request. This arrangement allowsvirtual warehouse resource manager 508 to distribute multiple requestsacross the virtual warehouses 510-514 based on efficiency, availableresources, and the availability of cached data within the virtualwarehouses 510-514. When determining how to route data processingrequests, virtual warehouse resource manager 508 considers availableresources, current resource loads, number of current users, and thelike.

In some embodiments, fault tolerance systems create a new virtualwarehouses in response to a failure of a virtual warehouse. The newvirtual warehouse may be in the same virtual warehouse group or may becreated in a different virtual warehouse group at a different geographiclocation.

Each virtual warehouse 510-514 is configured to communicate with asubset of all databases 518-528. For example, in environment 500,virtual warehouse 510 is configured to communicate with databases 518,520, and 526. Similarly, virtual warehouse 512 is configured tocommunicate with databases 520, 522, 524, and 528. And, virtualwarehouse 514 is configured to communicate with databases 520, 526, and528. In alternate embodiments, virtual warehouses 510-514 maycommunicate with any (or all) of the databases 518-528.

Although environment 500 shows one virtual warehouse group 516,alternate embodiments may include any number of virtual warehousegroups, each associated with any number of virtual warehouses. Forexample, different virtual warehouses may be created for each customeror group of users. Additionally, different virtual warehouses may becreated for different entities, or any other group accessing differentdata sets. Multiple virtual warehouse groups may have different sizesand configurations. The number of virtual warehouse groups in aparticular environment is dynamic and may change based on the changingneeds of the users and other systems in the environment.

FIG. 6 is a block diagram depicting another example operatingenvironment 600 having multiple distributed virtual warehouses andvirtual warehouse groups. Environment 600 includes resource manager 102that communicates with virtual warehouse groups 604 and 606 through adata communication network 602. Warehouse group 604 includes two virtualwarehouses 608 and 610, and warehouse group 606 includes another twovirtual warehouses 614 and 616. Resource manager 102 also communicateswith virtual warehouse 612 (which is not part of a virtual warehousegroup) through data communication network 602.

Virtual warehouse groups 604 and 606 as well as virtual warehouse 612communicate with databases 620, 622, and 624 through a datacommunication network 618. In some embodiments data communicationnetworks 602 and 618 are the same network. Environment 600 allowsresource manager 102 to coordinate user data storage and retrievalrequests across the multiple virtual warehouses 608-616 to store andretrieve data in databases 620-624. Virtual warehouse groups 604 and 606can be located in the same geographic area, or can be separatedgeographically. Additionally, virtual warehouse groups 604 and 606 canbe implemented by the same entity or by different entities.

The systems and methods described herein allow data to be stored andaccessed as a service that is separate from computing (or processing)resources. Even if no computing resources have been allocated from theexecution platform, data is available to a virtual warehouse withoutrequiring reloading of the data from a remote data source. Thus, data isavailable independently of the allocation of computing resourcesassociated with the data. The described systems and methods are usefulwith any type of data. In particular embodiments, data is stored in astructured, optimized format. The decoupling of the data storage/accessservice from the computing services also simplifies the sharing of dataamong different users and groups. As discussed herein, each virtualwarehouse can access any data to which it has access permissions, evenat the same time as other virtual warehouses are accessing the samedata. This architecture supports running queries without any actual datastored in the local cache. The systems and methods described herein arecapable of transparent dynamic data movement, which moves data from aremote storage device to a local cache, as needed, in a manner that istransparent to the user of the system. Further, this architecturesupports data sharing without prior data movement since any virtualwarehouse can access any data due to the decoupling of the data storageservice from the computing service.

FIG. 7 is a flow diagram depicting an embodiment of a method 700 formanaging data storage and retrieval operations. Initially, method 700receives a statement, request or query from a user at 702. A statementis any request or command to perform a data-related operation. Examplestatements include data retrieval requests, data storage requests, datatransfer requests, data queries, and the like. In some embodiments, thestatement is implemented as an SQL statement. A resource manager createsa query coordinator at 704 to manage the received statement. Forexample, the query coordinator manages the various tasks necessary toprocess the received statement, including interacting with an executionplatform and one or more data storage devices. In some embodiments, thequery coordinator is a temporary routine created specifically to managethe received statement.

Method 700 continues as the resource manager determines multiple tasksnecessary to process the received statement at 706. The multiple tasksmay include, for example, accessing data from a cache in an executionnode, retrieving data from a remote storage device, updating data in acache, storing data in a remote storage device, and the like. Theresource manager also distributes the multiple tasks to execution nodesin the execution platform at 708. As discussed herein, the executionnodes in the execution platform are implemented within virtualwarehouses. Each execution node performs an assigned task and returns atask result to the resource manager at 710. In some embodiments, theexecution nodes return the task results to the query coordinator. Theresource manager receives the multiple task results and creates astatement result at 712, and communicates the statement result to theuser at 714. In some embodiments, the query coordinator is deleted afterthe statement result is communicated to the user.

FIG. 8 is a flow diagram depicting an embodiment of a method 800 forprovisioning new virtual warehouses. Initially, a resource manager(e.g., resource manager 102) monitors data processing requests receivedfrom multiple users at 802. As discussed herein, these data processingrequests may include data search queries, data read requests, data writerequests, and the like. The resource manager also monitors currentresource utilization, query response rates, a number of usersinteracting with each virtual warehouse, and other performance metricsfor the existing virtual warehouses at 804. The current resourceutilization includes, for example, cache utilization and processorutilization in one or more execution nodes in the virtual warehouses.The current resource utilization may be expressed as a percentage ofmaximum utilization (e.g., 40% utilization of a cache and 60%utilization of a processor). The query response rates may be expressed,for example, as an average time required to respond to user queries. Insome embodiments, the resource manager also monitors a number of queriesrunning concurrently, percentage of maximum load, whether some queryprocessing need to be delayed, and the like.

Method 800 continues as the resource manager determines current andfuture resource needs at 806. For example, the resource manager canidentify pending data processing requests as well as expected requestsin the near future. The expected requests may be determined based onprevious patterns of previously received data processing requests fromparticular users at particular times. Additionally, the resource managermay receive advance notice of a data processing project and candetermine the resources needed to handle that project. The resourcemanager then determines at 808 whether one or more additional virtualwarehouses are needed based on the current data processing requests, thecurrent resource utilization, query response rates, and otherperformance metrics associated with the existing virtual warehouses. Ifan additional virtual warehouse is needed, the resource managerprovisions a new virtual warehouse at 810. For example, if the resourcemanager is aware of an upcoming data processing project that willrequire more resources than are currently available, the resourcemanager can decide to provision one or more new virtual warehouses tohandle the upcoming data processing project. The new virtual warehouseis provisioned quickly such that the new virtual warehouse is ready tohandle the data processing requests immediately at the start time of theproject.

In addition to adding new virtual warehouses, method 800 may determinewhether to deactivate one or more virtual warehouses at 812. If anyvirtual warehouses are no longer necessary, the resource managerdeactivates one or more virtual warehouses at 814. For example, if aparticular virtual warehouse was created for a specific data processingproject, that virtual warehouse may be deactivated after the specificproject is completed. Additionally, if many virtual warehouses areoperating with low resource utilization, some of the existing virtualwarehouses can be deactivated without degrading the performance of theremaining virtual warehouses. In some embodiments, a virtual warehouseis created for a specific time period. After the time period haselapsed, the virtual warehouse may be deactivated. In other embodiments,the resource manager identifies virtual warehouses that have been idlefor a particular amount of time and automatically deactivates thosevirtual warehouses.

In some situations, particular users (or system administrators) maydesire increased performance (e.g., increased query response time). Inthese situations, additional virtual warehouses may be added to supportthis increased performance. In other implementations, the resourcemanager predicts upcoming resource needs based on scheduled (but not yetexecuted) queries. If the scheduled queries will significantly degradethe system's performance, the resource manager can add more resourcesprior to executing those queries, thereby maintaining overall systemperformance. After those queries are executed, the added resources canbe deactivated by the resource manager.

In some embodiments, the resource manager predicts a time required toexecute a particular query (or group of queries). Based on current queryprocessing performance (e.g., query processing delays, systemutilization, etc.), the resource manager determines whether additionalresources are needed for that particular query or group of queries. Forexample, if the current query processing delay exceeds a thresholdvalue, the resource manager may create one or more new execution nodesto provide additional resources for processing the particular query orgroup of queries. After processing of the query or group of queries iscomplete, the resource manager may deactivate the new execution node(s)if they are no longer needed for processing other queries.

In some embodiments, a particular user may require certain performancelevels when processing the user's queries. For example, the user mayrequire a query response within a particular time period, such as 5seconds. In these embodiments, the resource manager may allocateadditional resources prior to executing the user's queries to ensure theuser's performance levels are achieved.

FIGS. 9A and 9B depict a flow diagram of an embodiment of a method 900for scaling resources. Initially, a resource manager monitors requestsreceived from multiple users at 902. Additionally, the resource managermonitors current data allocation in multiple storage devices (e.g.,multiple remote storage devices) and monitors current utilization ofmultiple processors at 904. Method 900 continues at 906 as the resourcemanager determines whether additional data storage capacity is neededbased on the data processing requests and the current data allocation inthe multiple storage devices. The resource manager allocates additionaldata storage resources to support the multiple users at 908 if itdetermines that additional data capacity is needed. This additional datacapacity may be accessible to any number of the multiple users, asdetermined by the resource manager. In some embodiments, the addition ofmore data capacity is referred to as “horizontal scaling.”

The resource manager also determines whether some of the allocated datacapacity is no longer needed at 910. If the resource manager determinesthat some of the allocated data capacity is no longer needed, theresource manager releases some of the data capacity at 912. For example,the released data capacity is returned to a pool of available datacapacity that becomes available for use by other systems or services.If, at a later time, additional data capacity is needed, the resourcemanager can access data resources from the available pool.

Method 900 continues at 914 as the resource manager determines whetheradditional processing resources are needed based on the data processingrequests and the current utilization of the multiple processors. Theresource manager allocates additional processing resources to supportthe multiple users at 916 if it determines that additional processingresources are needed. The additional processing resources may beaccessible to any number of the multiple users, as directed by theresource manager. For example, if a particular user has submitted alarge number of data queries, a portion of the additional processingresources may be assigned to that user to assist in processing the dataqueries. In some embodiments, the addition of more processing resourcesis referred to as “vertical scaling.”

The resource manager also determines whether some of the currentlyallocated processing resources are no longer needed at 918. The resourcemanager releases some of the processing resources that are no longerneeded at 920 if it determines that some of the currently allocatedprocessing resources are no longer needed. In some embodiments, thereleased processing resources are returned to a pool of availableprocessing resources that are available for use by other systems orservices. If additional processing resources are needed at a later time,the resource manager can access processing resources from the availablepool.

As described herein, data processing platform 100 supports the dynamicactivation and deactivation of various resources, such as data storagecapacity, processing resources, cache resources, and the like. Thesingle data processing platform 100 can be dynamically changed on-demandbased on the current data storage and processing requirements of thepending and anticipated data processing requests. As the data storageand processing requirements change, data processing platform 100automatically adjusts to maintain a substantially uniform level of dataprocessing performance.

Additionally, the described data processing platform 100 permits changesto the data storage capacity and the processing resources independently.For example, the data storage capacity can be modified without makingany changes to the existing processing resources. Similarly, theprocessing resources can be modified without making any changes to theexisting data storage capacity.

FIG. 10 is a flow diagram depicting an embodiment of another method forscaling resources. Initially, a resource manager (e.g., resource manager102) monitors requests received from multiple users at 1002. Asdiscussed herein, these requests may include data search queries, dataread requests, data write requests, and the like. The resource manageralso identifies a number of current users at 1004. This number changesregularly as new users access the system and existing users disconnectfrom the system. Additionally, the resource manager identifies activitylevels of the current users at 1006. The activity level includes, forexample, the number of queries submitted over a period of time, thenumber of pending queries, and the like. This activity level may bedetermined for each user, or may be represented as an average activityfor all of the current users (or groups of current users).

Based on the number of current users and the activity level of thecurrent users, the resource manager determines data storage resourcesand processing resources needed to support the activity level of thecurrent users at 1008. Since the number of users is changing regularly,and the user activity levels may change frequently, the resource managercontinuously determines whether the current resources adequately supportthe current users. If additional resources are needed at 1010, theresource manager provisions one or more new virtual warehouses at 1012to support the current users. Similarly, if the number of users and/orthe activity level decreases, the resource manager may deactivate one ormore virtual warehouses if they are no longer needed to support thecurrent users.

In some implementations, the same file is cached by multiple executionnodes at the same time. This multiple caching of files helps with loadbalancing (e.g., balancing data processing tasks) across multipleexecution nodes. Additionally, caching a file in multiple executionnodes helps avoid potential bottlenecks when significant amounts of dataare trying to pass through the same communication link. Thisimplementation also supports the parallel processing of the same data bydifferent execution nodes.

The systems and methods described herein take advantage of the benefitsof both shared-disk systems and the shared-nothing architecture. Thedescribed platform for storing and retrieving data is scalable like theshared-nothing architecture once data is cached locally. It also has allthe benefits of a shared-disk architecture where processing nodes can beadded and removed without any constraints (e.g., for 0 to N) and withoutrequiring any explicit reshuffling of data.

FIG. 11 is a block diagram depicting an example computing device 1100.In some embodiments, computing device 1100 is used to implement one ormore of the systems and components discussed herein. For example,computing device 1100 may allow a user or administrator to accessresource manager 102. Further, computing device 1100 may interact withany of the systems and components described herein. Accordingly,computing device 1100 may be used to perform various procedures andtasks, such as those discussed herein. Computing device 1100 canfunction as a server, a client or any other computing entity. Computingdevice 1100 can be any of a wide variety of computing devices, such as adesktop computer, a notebook computer, a server computer, a handheldcomputer, a tablet, and the like.

Computing device 1100 includes one or more processor(s) 1102, one ormore memory device(s) 1104, one or more interface(s) 1106, one or moremass storage device(s) 1108, and one or more Input/Output (I/O)device(s) 1110, all of which are coupled to a bus 1112. Processor(s)1102 include one or more processors or controllers that executeinstructions stored in memory device(s) 1104 and/or mass storagedevice(s) 1108. Processor(s) 1102 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 1104 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM)) and/or nonvolatilememory (e.g., read-only memory (ROM)). Memory device(s) 1104 may alsoinclude rewritable ROM, such as Flash memory.

Mass storage device(s) 1108 include various computer readable media,such as magnetic tapes, magnetic disks, optical disks, solid statememory (e.g., Flash memory), and so forth. Various drives may also beincluded in mass storage device(s) 1108 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)1108 include removable media and/or non-removable media.

I/O device(s) 1110 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 1100.Example I/O device(s) 1110 include cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Interface(s) 1106 include various interfaces that allow computing device1100 to interact with other systems, devices, or computing environments.Example interface(s) 1106 include any number of different networkinterfaces, such as interfaces to local area networks (LANs), wide areanetworks (WANs), wireless networks, and the Internet.

Bus 1112 allows processor(s) 1102, memory device(s) 1104, interface(s)1106, mass storage device(s) 1108, and I/O device(s) 1110 to communicatewith one another, as well as other devices or components coupled to bus1112. Bus 1112 represents one or more of several types of busstructures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, andso forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 1100, and areexecuted by processor(s) 1102. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein.

Although the present disclosure is described in terms of certainpreferred embodiments, other embodiments will be apparent to those ofordinary skill in the art, given the benefit of this disclosure,including embodiments that do not provide all of the benefits andfeatures set forth herein, which are also within the scope of thisdisclosure. It is to be understood that other embodiments may beutilized, without departing from the scope of the present disclosure.

What is claimed is:
 1. A method comprising: receiving a set of queriescomprising one or more queries to be processed by a virtual datawarehouse; provisioning a set of nodes from a first plurality of nodes,the set of nodes to process the set of queries, wherein the set of nodesare coupled to a set of storage resources shared by each of the set ofnodes to store data for the set of queries; monitoring a utilization ofthe set of nodes as the set of nodes processes the set of queries;updating a number of the nodes in the set of nodes provisioned based onthe utilization of the set of nodes as the set of nodes that areprocessing the set of queries, wherein updating of the set of storageresources is independent to a change in the number of nodes; andprocessing the set of queries using the updated set of nodes.
 2. Themethod of claim 1, wherein the updating the number of nodes includesincreasing the number of nodes in the set of nodes.
 3. The method ofclaim 1, wherein the updating the number of nodes includes decreasingthe number of nodes in the set of nodes.
 4. The method of claim 1,wherein the utilization is selected from a group consisting of a numberof queries running concurrently, a percentage of maximum load, and queryprocessing delay.
 5. The method of claim 1, further comprising:receiving an additional query for the set of queries.
 6. The method ofclaim 5, determining a change in the set of nodes in response toreceiving the additional query.
 7. The method of claim 1, furthercomprising: optimizing at least one of the queries in the set ofqueries.
 8. The method of claim 1, wherein the virtual data warehouse iscoupled to a relational database.
 9. The method of claim 8, wherein therelational database is a structured query language database.
 10. Themethod of claim 1, wherein the virtual data warehouse is coupled to adatabase system that is a multi-tenant database and isolates computingresources and data between a plurality of different customers.
 11. Asystem comprising: a set of storage resources; a resource managerimplemented in a processor, which executes to: receive a set of queriescomprising one or more queries to be processed by a virtual datawarehouse; provision a set of nodes from a first plurality of nodes, theset of nodes to process the set of queries, wherein the set of nodes arecoupled to a set of a set of storage resources shared by each of the setof nodes to store data for the set of queries; monitor a utilization ofthe set of nodes as the set of nodes processes the set of queries; andupdate a number of the nodes in the set of nodes provisioned based onthe utilization of the set of nodes as the set of nodes that areprocessing the set of queries, wherein updating of the set of storageresources is independent to a change in the number of nodes; and aplurality of nodes to process the set of queries using the updated setof nodes.
 12. The system of claim 11, wherein the resource managerupdates the number of nodes by increasing the number of nodes in the setof nodes.
 13. The system of claim 11, wherein the resource managerupdates the number of nodes by decreasing the number of nodes in the setof nodes.
 14. The system of claim 11, wherein the utilization isselected from a group consisting of a number of queries runningconcurrently, a percentage of maximum load, and query processing delay.15. The system of claim 11, wherein the resource manager further toreceive an additional query for the set of queries.
 16. The system ofclaim 15, wherein the resource manager determines a change in the set ofnodes in response to receiving the additional query.
 17. The system ofclaim 11, wherein the resource manager further to optimize at least oneof the queries in the set of queries.
 18. The system of claim 11,wherein the virtual data warehouse is coupled to a relational database.19. The system of claim 18, wherein the relational database is astructured query language database.
 20. The system of claim 11, whereinthe virtual data warehouse is coupled to a database system that is amulti-tenant database and isolates computing resources and data betweena plurality of different customers.
 21. A non-transitorycomputer-readable medium storing instructions which, when executed byone or more second nodes of a computing device, cause the one or moresecond nodes to: receive a set of queries comprising one or more queriesto be processed by a virtual data warehouse; provision a set of nodesfrom a first plurality of nodes, the set of nodes to process the set ofqueries, wherein the set of nodes are coupled to a set of a set ofstorage resources shared by each of the set of nodes to store data forthe set of queries; monitor a utilization of the set of nodes as the setof nodes processes the set of queries; update a number of the nodes inthe set of nodes provisioned based on the utilization of the set ofnodes as the set of nodes that are processing the set of queries,wherein updating of the set of storage resources is independent to achange in the number of nodes; and process the set of queries using theupdated set of nodes.
 22. The non-transitory computer-readable medium ofclaim 21, wherein the updating the number of nodes includes increasingthe number of nodes in the set of nodes.
 23. The non-transitorycomputer-readable medium of claim 21, wherein the updating the number ofnodes includes decreasing the number of nodes in the set of nodes.